{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Face Recognition for the Happy House\n",
    "\n",
    "We are building a face recognition. Many of the ideas presented here are from [FaceNet](https://arxiv.org/pdf/1503.03832.pdf) and [DeepFace](https://research.fb.com/wp-content/uploads/2016/11/deepface-closing-the-gap-to-human-level-performance-in-face-verification.pdf). \n",
    "\n",
    "Face recognition problems commonly fall into two categories: \n",
    "\n",
    "- **Face Verification** - \"is this the claimed person?\". For example, at some airports, you can pass through customs by letting a system scan your passport and then verifying that you (the person carrying the passport) are the correct person. A mobile phone that unlocks using your face is also using face verification. This is a 1:1 matching problem. \n",
    "- **Face Recognition** - \"who is this person?\". Not needing to otherwise identify oneself. This is a 1:K matching problem. \n",
    "\n",
    "FaceNet learns a neural network that encodes a face image into a vector of 128 numbers. By comparing two such vectors, you can then determine if two pictures are of the same person.\n",
    "    \n",
    "In this notebook I will be performing the following\n",
    "- Using a pretrained model to map face images into 128-dimensional encodings\n",
    "- Using these encodings to perform face verification and face recognition\n",
    "\n",
    "I will be using a pre-trained model which represents ConvNet activations using a \"channels first\" convention, as opposed to the \"channels last\" convention used in lecture and previous programming assignments. In other words, a batch of images will be of shape $(m, n_C, n_H, n_W)$ instead of $(m, n_H, n_W, n_C)$. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The autoreload extension is already loaded. To reload it, use:\n",
      "  %reload_ext autoreload\n"
     ]
    }
   ],
   "source": [
    "#Import Required Packages\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Conv2D, ZeroPadding2D, Activation, Input, concatenate\n",
    "from keras.models import Model\n",
    "from keras.layers.normalization import BatchNormalization\n",
    "from keras.layers.pooling import MaxPooling2D, AveragePooling2D\n",
    "from keras.layers.merge import Concatenate\n",
    "from keras.layers.core import Lambda, Flatten, Dense\n",
    "from keras.initializers import glorot_uniform\n",
    "from keras.engine.topology import Layer\n",
    "from keras import backend as K\n",
    "K.set_image_data_format('channels_first')\n",
    "import cv2\n",
    "import os\n",
    "import numpy as np\n",
    "from numpy import genfromtxt\n",
    "import pandas as pd\n",
    "import tensorflow as tf\n",
    "from fr_utils import *\n",
    "from inception_blocks_v2 import *\n",
    "\n",
    "%matplotlib inline\n",
    "%load_ext autoreload\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Naive Face Verificaiton\n",
    "\n",
    "When given two images and you have to tell if they are of the same person. The simplest way to do this is to compare the two images pixel-by-pixel. If the distance between the raw images are less than a chosen threshold, it may be the same person!\n",
    "\n",
    "<img src=\"images/pixel_comparison.png\" style=\"width:380px;height:150px;\">\n",
    "<caption><center> <u> <font color='purple'> **Figure 1** </u></center></caption>\n",
    "    \n",
    "Of course, this performs really poorly, since the pixel values change dramatically due to variations in lighting, orientation of the person's face, even minor changes in head position, and so on.\n",
    "\n",
    "Instead of using a raw image, an image can be encoded so that element-wise comparisons of the encoding gives more accurate judgments as to whethe two pictures are of the same person"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Encoding Face Images into 128-dimensional vector\n",
    "\n",
    "FaceNet takes a lot of data and a long time to train. Instead of retraining the model, I will load the pre-trained weights. The network architecture file inception_blocks.py is provided by deeplearning.ai follows the Inception model from [Szegedy *et al.*](https://arxiv.org/abs/1409.4842)\n",
    "\n",
    "- This network uses 96x96 dimensional RGB images as its input. Specifically, inputs a face image (or batch of $m$ face images) as a tensor of shape $(m, n_C, n_H, n_W) = (m, 3, 96, 96)$ \n",
    "- It outputs a matrix of shape $(m, 128)$ that encodes each input face image into a 128-dimensional vector\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Produce Model\n",
    "FRmodel = faceRecoModel(input_shape=(3, 96, 96))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total Params: 3743280\n"
     ]
    }
   ],
   "source": [
    "print(\"Total Params:\", FRmodel.count_params())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By using a 128-neuron fully connected layer as its last layer, the model ensures that the output is an encoding vector of size 128. You then use the encodings the compare two face images as follows:\n",
    "\n",
    "<img src=\"images/distance_kiank.png\" style=\"width:680px;height:250px;\">\n",
    "<caption><center> <u> <font color='purple'> **Figure 2**: <br> </u> <font color='purple'> By computing a distance between two encodings and thresholding, you can determine if the two pictures represent the same person</center></caption>\n",
    "\n",
    "So, an encoding is a good one if: \n",
    "- The encodings of two images of the same person are quite similar to each other \n",
    "- The encodings of two images of different persons are very different"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model training using the Triplet Loss\n",
    "Once we obtain the 128bit encoding of an image, we will use triplets of an image to calculate the triplet loss function. The triplet loss function trie's to push the encodings of two images of the same person (Anchor and Positive) closer together, while \"pulling\" the encodings of two images of different persons (Anchor and Negative) further apart. \n",
    "\n",
    "Training will use triplets of images $(A, P, N)$:  \n",
    "\n",
    "- A is an \"Anchor\" image--a picture of a person. \n",
    "- P is a \"Positive\" image--a picture of the same person as the Anchor image.\n",
    "- N is a \"Negative\" image--a picture of a different person than the Anchor image.\n",
    "\n",
    "These triplets are picked from our training dataset. We will write $(A^{(i)}, P^{(i)}, N^{(i)})$ to denote the $i$-th training example. \n",
    "\n",
    "You'd like to make sure that an image $A^{(i)}$ of an individual is closer to the Positive $P^{(i)}$ than to the Negative image $N^{(i)}$) by at least a margin $\\alpha$:\n",
    "\n",
    "$$\\mid \\mid f(A^{(i)}) - f(P^{(i)}) \\mid \\mid_2^2 + \\alpha < \\mid \\mid f(A^{(i)}) - f(N^{(i)}) \\mid \\mid_2^2$$\n",
    "\n",
    "You would thus like to minimize the following \"triplet cost\":\n",
    "\n",
    "$$\\mathcal{J} = \\sum^{m}_{i=1} \\large[ \\small \\underbrace{\\mid \\mid f(A^{(i)}) - f(P^{(i)}) \\mid \\mid_2^2}_\\text{(1)} - \\underbrace{\\mid \\mid f(A^{(i)}) - f(N^{(i)}) \\mid \\mid_2^2}_\\text{(2)} + \\alpha \\large ] \\small_+ \\tag{3}$$\n",
    "\n",
    "Here, we are using the notation \"$[z]_+$\" to denote $max(z,0)$.  \n",
    "\n",
    "Notes:\n",
    "- The term (1) is the squared distance between the anchor \"A\" and the positive \"P\" for a given triplet; you want this to be small. \n",
    "- The term (2) is the squared distance between the anchor \"A\" and the negative \"N\" for a given triplet, you want this to be relatively large, so it thus makes sense to have a minus sign preceding it. \n",
    "- $\\alpha$ is called the margin. It is a hyperparameter that you should pick manually. We will use $\\alpha = 0.2$. \n",
    "\n",
    "Most implementations also normalize the encoding vectors  to have norm equal one (i.e., $\\mid \\mid f(img)\\mid \\mid_2$=1); you won't have to worry about that here.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Triplet_Loss Function\n",
    "def triplet_loss(y_true, y_pred, alpha = 0.2):\n",
    "    \"\"\"\n",
    "    Implementation of the triplet loss as defined above\n",
    "    \n",
    "    Arguments:\n",
    "    y_true -- true labels, required when you define a loss in Keras, not needed in this function\n",
    "    y_pred -- python list containing three objects:\n",
    "            anchor -- the encodings for the anchor images, of shape (None, 128)\n",
    "            positive -- the encodings for the positive images, of shape (None, 128)\n",
    "            negative -- the encodings for the negative images, of shape (None, 128)\n",
    "    \n",
    "    Returns:\n",
    "    loss -- real number, value of the loss\n",
    "    \"\"\"\n",
    "    \n",
    "    anchor, positive, negative = y_pred[0], y_pred[1], y_pred[2]\n",
    "    \n",
    "    # Computing distance between anchor and the positive\n",
    "    pos_dist = tf.reduce_sum(tf.square(tf.subtract(y_pred[0],y_pred[1])), axis=-1)\n",
    "    \n",
    "    # Computing distance between anchor and the negative\n",
    "    neg_dist = tf.reduce_sum(tf.square(tf.subtract(y_pred[0],y_pred[2])), axis=-1)\n",
    "    \n",
    "    # Compute the difference between the two distances and add alpha\n",
    "    basic_loss = tf.add(tf.subtract(pos_dist, neg_dist), alpha)\n",
    "    \n",
    "    # Compute the maximum of basic_loss and 0.0. Sum over the training examples.\n",
    "    loss = tf.reduce_sum(tf.maximum(basic_loss, 0.0))\n",
    "    \n",
    "    ### END CODE HERE ###\n",
    "    \n",
    "    return loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss = 528.1427\n"
     ]
    }
   ],
   "source": [
    "with tf.Session() as test:\n",
    "    tf.set_random_seed(1)\n",
    "    y_true = (None, None, None)\n",
    "    y_pred = (tf.random_normal([3, 128], mean=6, stddev=0.1, seed = 1),\n",
    "              tf.random_normal([3, 128], mean=1, stddev=1, seed = 1),\n",
    "              tf.random_normal([3, 128], mean=3, stddev=4, seed = 1))\n",
    "    loss = triplet_loss(y_true, y_pred)\n",
    "    \n",
    "    print(\"loss = \" + str(loss.eval()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2 - Loading the trained model\n",
    "\n",
    "FaceNet is trained by minimizing the triplet loss. But since training requires a lot of data and a lot of computation, we won't train it from scratch here. Instead, we load a previously trained model. Load a model using the following cell; this might take a couple of minutes to run. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "W0810 17:26:42.801910 140406929377088 deprecation_wrapper.py:119] From /home/mohammad/anaconda3/envs/dl_detection/lib/python3.7/site-packages/keras/optimizers.py:790: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "FRmodel.compile(optimizer = 'adam', loss = triplet_loss, metrics = ['accuracy'])\n",
    "load_weights_from_FaceNet(FRmodel)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here're some examples of distances between the encodings between three individuals:\n",
    "\n",
    "<img src=\"images/distance_matrix.png\" style=\"width:380px;height:200px;\">\n",
    "<br>\n",
    "<caption><center> <u> <font color='purple'> **Figure 4**:</u> <br>  <font color='purple'> Example of distance outputs between three individuals' encodings</center></caption>\n",
    "\n",
    "Let's now use this model to perform face verification and face recognition! "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Building a database of individuals allowed to enter via facial recognition. \n",
    "To generate the encoding we use img_to_encoding(image_path, model) -> running the forward propogation of the model on the specified image. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Python dictionary of allowed individuals\n",
    "database = {}\n",
    "database[\"danielle\"] = img_to_encoding(\"images/danielle.png\", FRmodel)\n",
    "database[\"younes\"] = img_to_encoding(\"images/younes.jpg\", FRmodel)\n",
    "database[\"tian\"] = img_to_encoding(\"images/tian.jpg\", FRmodel)\n",
    "database[\"kian\"] = img_to_encoding(\"images/kian.jpg\", FRmodel)\n",
    "database[\"dan\"] = img_to_encoding(\"images/dan.jpg\", FRmodel)\n",
    "database[\"sebastiano\"] = img_to_encoding(\"images/sebastiano.jpg\", FRmodel)\n",
    "database[\"bertrand\"] = img_to_encoding(\"images/bertrand.jpg\", FRmodel)\n",
    "database[\"kevin\"] = img_to_encoding(\"images/kevin.jpg\", FRmodel)\n",
    "database[\"felix\"] = img_to_encoding(\"images/felix.jpg\", FRmodel)\n",
    "database[\"benoit\"] = img_to_encoding(\"images/benoit.jpg\", FRmodel)\n",
    "database[\"arnaud\"] = img_to_encoding(\"images/arnaud.jpg\", FRmodel)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Applying the model\n",
    "When someone scans their ID card providing their name, we will look up their encoding in the database, and use it to check if the person standing at the front door matches the name on the ID. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The verify() function will check if the picture (image_path) is actually the person called \"identity\". Utilizing the following steps\n",
    "\n",
    "* Compute the encoding of the image from image_path\n",
    "* Compute the distance about this encoding and the encoding of the identity image stored in the database\n",
    "* Open the door if the distance is less than 0.7, else do not open."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.linalg.norm\n",
    "def verify(image_path, identity, database, model):\n",
    "    \"\"\"\n",
    "    Function that verifies if the person on the \"image_path\" image is \"identity\".\n",
    "    \n",
    "    Arguments:\n",
    "    image_path -- path to an image\n",
    "    identity -- string, name of the person you'd like to verify the identity. Has to be a resident of the Happy house.\n",
    "    database -- python dictionary mapping names of allowed people's names (strings) to their encodings (vectors).\n",
    "    model -- your Inception model instance in Keras\n",
    "    \n",
    "    Returns:\n",
    "    dist -- distance between the image_path and the image of \"identity\" in the database.\n",
    "    door_open -- True, if the door should open. False otherwise.\n",
    "    \"\"\"\n",
    "        \n",
    "    # Compute the encoding for the image. Using img_to_encoding() \n",
    "    encoding = img_to_encoding(image_path, model)\n",
    "    \n",
    "    # Compute distance with identity's image\n",
    "    dist = float(np.linalg.norm(encoding - database[identity]))\n",
    "    \n",
    "    # Open the door if dist < 0.7, else don't open\n",
    "    if dist < 0.7:\n",
    "        print(\"It's \" + str(identity) + \", welcome home!\")\n",
    "        door_open = True\n",
    "    else:\n",
    "        print(\"It's not \" + str(identity) + \", please go away\")\n",
    "        door_open = False\n",
    "                \n",
    "    return dist, door_open"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Younes is trying to enter and the camera takes a picture of him (\"images/camera_0.jpg\"). Test if he's allowed to enter\n",
    "\n",
    "<img src=\"images/camera_0.jpg\" style=\"width:100px;height:100px;\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "It's younes, welcome home!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(0.6593842506408691, True)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "verify(\"images/camera_0.jpg\", \"younes\", database, FRmodel)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Benoit has been banned and removed from the database. He stole Kian's ID card and came back to the house to try to present himself as Kian. The front-door camera took a picture of Benoit (\"images/camera_2.jpg). Test if he's allowed to enter\n",
    "\n",
    "<img src=\"images/camera_2.jpg\" style=\"width:100px;height:100px;\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "It's not kian, please go away\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(0.8622572422027588, False)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "verify(\"images/camera_2.jpg\", \"kian\", database, FRmodel)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Facial Recognition - No ID or Name GIven\n",
    "\n",
    "The above works great, however, if someone uses their ID they cannot gain access. Instead of requesting for ID, the model will recognize the individual and allow them access (No ID or Name Required) using the who_is_it() function\n",
    "\n",
    "The approach is as follows:\n",
    "1. Compute the target encoding of the image from image_path\n",
    "2. Find the encoding from the database that has smallest distance with the target encoding. \n",
    "    - Initialize the `min_dist` variable to a large enough number (100). It will help you keep track of what is the closest encoding to the input's encoding.\n",
    "    - Loop over the database dictionary's names and encodings. To loop use `for (name, db_enc) in database.items()`.\n",
    "        - Compute L2 distance between the target \"encoding\" and the current \"encoding\" from the database.\n",
    "        - If this distance is less than the min_dist, then set min_dist to dist, and identity to name.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "def who_is_it(image_path, database, model):\n",
    "    \"\"\"\n",
    "    Implements face recognition by finding who is the person on the image_path image.\n",
    "    \n",
    "    Arguments:\n",
    "    image_path -- path to an image\n",
    "    database -- database containing image encodings along with the name of the person on the image\n",
    "    model -- your Inception model instance in Keras\n",
    "    \n",
    "    Returns:\n",
    "    min_dist -- the minimum distance between image_path encoding and the encodings from the database\n",
    "    identity -- string, the name prediction for the person on image_path\n",
    "    \"\"\"   \n",
    "    \n",
    "    #Compute the target \"encoding\" for the image. Use img_to_encoding() \n",
    "    encoding = img_to_encoding(image_path, model)\n",
    "    \n",
    "    #Find the closest encoding ##\n",
    "    \n",
    "    # Initialize \"min_dist\" to a large value, say 100 \n",
    "    min_dist = 100\n",
    "    \n",
    "    # Loop over the database dictionary's names and encodings.\n",
    "    for (name, db_enc) in database.items():\n",
    "        \n",
    "        # Compute L2 distance between the target \"encoding\" and the current \"emb\" from the database.\n",
    "        dist = np.linalg.norm(encoding - db_enc)\n",
    "\n",
    "        # If this distance is less than the min_dist, then set min_dist to dist, and identity to name.\n",
    "        if dist < min_dist:\n",
    "            min_dist = dist\n",
    "            identity = name\n",
    "    \n",
    "    if min_dist > 0.7:\n",
    "        print(\"Not in the database. Access Denied!\")\n",
    "    else:\n",
    "        print (\"It's \" + str(identity) + \", Access Granted!\")\n",
    "        \n",
    "    return"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "It's younes, Access Granted!\n"
     ]
    }
   ],
   "source": [
    "who_is_it(\"images/camera_0.jpg\", database, FRmodel)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### References:\n",
    "\n",
    "- Florian Schroff, Dmitry Kalenichenko, James Philbin (2015). [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://arxiv.org/pdf/1503.03832.pdf)\n",
    "- Yaniv Taigman, Ming Yang, Marc'Aurelio Ranzato, Lior Wolf (2014). [DeepFace: Closing the gap to human-level performance in face verification](https://research.fb.com/wp-content/uploads/2016/11/deepface-closing-the-gap-to-human-level-performance-in-face-verification.pdf) \n",
    "- The pretrained model we use is inspired by Victor Sy Wang's implementation and was loaded using his code: https://github.com/iwantooxxoox/Keras-OpenFace.\n",
    "- Our implementation also took a lot of inspiration from the official FaceNet github repository: https://github.com/davidsandberg/facenet \n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "deeplearning_detection",
   "language": "python",
   "name": "dl_detection"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
